Make LSP notebooks event also trigger for those who may be on the control group#19194
Merged
Make LSP notebooks event also trigger for those who may be on the control group#19194
Conversation
… group of notebooks LSP experiment
debonte
approved these changes
May 25, 2022
kimadeline
approved these changes
May 25, 2022
wesm
pushed a commit
to posit-dev/positron
that referenced
this pull request
Mar 28, 2024
… group of notebooks LSP experiment (microsoft/vscode-python#19194)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
To have more sensitive scorecards for the LSP notebooks experiment, I wanted to be able to add a segment for users who meet all the criteria of the experiment (i.e. those who are truly eligible to experience LSP notebooks).
I noticed the telemetry event we had (
python_experiments_lsp_notebooks) is only triggering for those who have thepylanceLspNotebooksEnabledsetting set totrue. So people in the control group who haven't touched this setting won't trigger this event. If I were to use it as segment in the scorecard, it would give me an SRM (sample ratio mismatch), because only people in treatment would trigger it.With this change, my intent is to make this event also be triggered for those who may be on the control group (so not necessarily experiencing notebooks LSP, but meeting all the criteria for it). A few of these people may have the setting set to true, and therefore be in the control group but experiencing notebooks LSP, which is noisy data, but I expect the # of people who manually flip that setting to be low and therefore am not too concerned about it :)